{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第六课 Seq2Seq, Attention\n",
    "\n",
    "褚则伟 zeweichu@gmail.com\n",
    "\n",
    "在这份notebook当中，我们会(尽可能)复现Luong的attention模型\n",
    "\n",
    "由于我们的数据集非常小，只有一万多个句子的训练数据，所以训练出来的模型效果并不好。如果大家想训练一个好一点的模型，可以参考下面的资料。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 更多阅读\n",
    "\n",
    "#### 课件\n",
    "- [cs224d](http://cs224d.stanford.edu/lectures/CS224d-Lecture15.pdf)\n",
    "\n",
    "\n",
    "#### 论文\n",
    "- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078)\n",
    "- [Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025?context=cs)\n",
    "- [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1406.1078)\n",
    "\n",
    "\n",
    "#### PyTorch代码\n",
    "- [seq2seq-tutorial](https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb)\n",
    "- [Tutorial from Ben Trevett](https://github.com/bentrevett/pytorch-seq2seq)\n",
    "- [IBM seq2seq](https://github.com/IBM/pytorch-seq2seq)\n",
    "- [OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py)\n",
    "\n",
    "\n",
    "#### 更多关于Machine Translation\n",
    "- [Beam Search](https://www.coursera.org/lecture/nlp-sequence-models/beam-search-4EtHZ)\n",
    "- Pointer network 文本摘要\n",
    "- Copy Mechanism 文本摘要\n",
    "- Converage Loss \n",
    "- ConvSeq2Seq\n",
    "- Transformer\n",
    "- Tensor2Tensor\n",
    "\n",
    "#### TODO\n",
    "- 建议同学尝试对中文进行分词\n",
    "\n",
    "#### NER\n",
    "- https://github.com/allenai/allennlp/tree/master/allennlp\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import math\n",
    "from collections import Counter\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读入中英文数据\n",
    "- 英文我们使用nltk的word tokenizer来分词，并且使用小写字母\n",
    "- 中文我们直接使用单个汉字作为基本单元"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_data(in_file):\n",
    "    cn = []\n",
    "    en = []\n",
    "    num_examples = 0\n",
    "    with open(in_file, 'r') as f:\n",
    "        for line in f:\n",
    "            line = line.strip().split(\"\\t\")\n",
    "            \n",
    "            en.append([\"BOS\"] + nltk.word_tokenize(line[0].lower()) + [\"EOS\"])\n",
    "            # split chinese sentence into characters\n",
    "            cn.append([\"BOS\"] + [c for c in line[1]] + [\"EOS\"])\n",
    "    return en, cn\n",
    "\n",
    "train_file = \"nmt/en-cn/train.txt\"\n",
    "dev_file = \"nmt/en-cn/dev.txt\"\n",
    "train_en, train_cn = load_data(train_file)\n",
    "dev_en, dev_cn = load_data(dev_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "构建单词表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "UNK_IDX = 0\n",
    "PAD_IDX = 1\n",
    "def build_dict(sentences, max_words=50000):\n",
    "    word_count = Counter()\n",
    "    for sentence in sentences:\n",
    "        for s in sentence:\n",
    "            word_count[s] += 1\n",
    "    ls = word_count.most_common(max_words)\n",
    "    total_words = len(ls) + 2\n",
    "    word_dict = {w[0]: index+2 for index, w in enumerate(ls)}\n",
    "    word_dict[\"UNK\"] = UNK_IDX\n",
    "    word_dict[\"PAD\"] = PAD_IDX\n",
    "    return word_dict, total_words\n",
    "\n",
    "en_dict, en_total_words = build_dict(train_en)\n",
    "cn_dict, cn_total_words = build_dict(train_cn)\n",
    "inv_en_dict = {v: k for k, v in en_dict.items()}\n",
    "inv_cn_dict = {v: k for k, v in cn_dict.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把单词全部转变成数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encode(en_sentences, cn_sentences, en_dict, cn_dict, sort_by_len=True):\n",
    "    '''\n",
    "        Encode the sequences. \n",
    "    '''\n",
    "    length = len(en_sentences)\n",
    "    out_en_sentences = [[en_dict.get(w, 0) for w in sent] for sent in en_sentences]\n",
    "    out_cn_sentences = [[cn_dict.get(w, 0) for w in sent] for sent in cn_sentences]\n",
    "\n",
    "    # sort sentences by english lengths\n",
    "    def len_argsort(seq):\n",
    "        return sorted(range(len(seq)), key=lambda x: len(seq[x]))\n",
    "       \n",
    "    # 把中文和英文按照同样的顺序排序\n",
    "    if sort_by_len:\n",
    "        sorted_index = len_argsort(out_en_sentences)\n",
    "        out_en_sentences = [out_en_sentences[i] for i in sorted_index]\n",
    "        out_cn_sentences = [out_cn_sentences[i] for i in sorted_index]\n",
    "        \n",
    "    return out_en_sentences, out_cn_sentences\n",
    "\n",
    "train_en, train_cn = encode(train_en, train_cn, en_dict, cn_dict)\n",
    "dev_en, dev_cn = encode(dev_en, dev_cn, en_dict, cn_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BOS 他 来 这 里 的 目 的 是 什 么 ？ EOS\n",
      "BOS for what purpose did he come here ? EOS\n"
     ]
    }
   ],
   "source": [
    "# train_cn[:10]\n",
    "k = 10000\n",
    "print(\" \".join([inv_cn_dict[i] for i in train_cn[k]]))\n",
    "print(\" \".join([inv_en_dict[i] for i in train_en[k]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把全部句子分成batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_minibatches(n, minibatch_size, shuffle=True):\n",
    "    idx_list = np.arange(0, n, minibatch_size) # [0, 1, ..., n-1]\n",
    "    if shuffle:\n",
    "        np.random.shuffle(idx_list)\n",
    "    minibatches = []\n",
    "    for idx in idx_list:\n",
    "        minibatches.append(np.arange(idx, min(idx + minibatch_size, n)))\n",
    "    return minibatches\n",
    "\n",
    "def prepare_data(seqs):\n",
    "    lengths = [len(seq) for seq in seqs]\n",
    "    n_samples = len(seqs)\n",
    "    max_len = np.max(lengths)\n",
    "\n",
    "    x = np.zeros((n_samples, max_len)).astype('int32')\n",
    "    x_lengths = np.array(lengths).astype(\"int32\")\n",
    "    for idx, seq in enumerate(seqs):\n",
    "        x[idx, :lengths[idx]] = seq\n",
    "    return x, x_lengths #x_mask\n",
    "\n",
    "def gen_examples(en_sentences, cn_sentences, batch_size):\n",
    "    minibatches = get_minibatches(len(en_sentences), batch_size)\n",
    "    all_ex = []\n",
    "    for minibatch in minibatches:\n",
    "        mb_en_sentences = [en_sentences[t] for t in minibatch]\n",
    "        mb_cn_sentences = [cn_sentences[t] for t in minibatch]\n",
    "        mb_x, mb_x_len = prepare_data(mb_en_sentences)\n",
    "        mb_y, mb_y_len = prepare_data(mb_cn_sentences)\n",
    "        all_ex.append((mb_x, mb_x_len, mb_y, mb_y_len))\n",
    "    return all_ex\n",
    "\n",
    "batch_size = 64\n",
    "train_data = gen_examples(train_en, train_cn, batch_size)\n",
    "random.shuffle(train_data)\n",
    "dev_data = gen_examples(dev_en, dev_cn, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 没有Attention的版本\n",
    "下面是一个更简单的没有Attention的encoder decoder模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class PlainEncoder(nn.Module):\n",
    "    def __init__(self, vocab_size, hidden_size, dropout=0.2):\n",
    "        super(PlainEncoder, self).__init__()\n",
    "        self.embed = nn.Embedding(vocab_size, hidden_size)\n",
    "        self.rnn = nn.GRU(hidden_size, hidden_size, batch_first=True)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x, lengths):\n",
    "        sorted_len, sorted_idx = lengths.sort(0, descending=True)\n",
    "        x_sorted = x[sorted_idx.long()]\n",
    "        embedded = self.dropout(self.embed(x_sorted))\n",
    "        \n",
    "        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, sorted_len.long().cpu().data.numpy(), batch_first=True)\n",
    "        packed_out, hid = self.rnn(packed_embedded)\n",
    "        out, _ = nn.utils.rnn.pad_packed_sequence(packed_out, batch_first=True)\n",
    "        _, original_idx = sorted_idx.sort(0, descending=False)\n",
    "        out = out[original_idx.long()].contiguous()\n",
    "        hid = hid[:, original_idx.long()].contiguous()\n",
    "        \n",
    "        return out, hid[[-1]]\n",
    "\n",
    "class PlainDecoder(nn.Module):\n",
    "    def __init__(self, vocab_size, hidden_size, dropout=0.2):\n",
    "        super(PlainDecoder, self).__init__()\n",
    "        self.embed = nn.Embedding(vocab_size, hidden_size)\n",
    "        self.rnn = nn.GRU(hidden_size, hidden_size, batch_first=True)\n",
    "        self.out = nn.Linear(hidden_size, vocab_size)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, y, y_lengths, hid):\n",
    "        sorted_len, sorted_idx = y_lengths.sort(0, descending=True)\n",
    "        y_sorted = y[sorted_idx.long()]\n",
    "        hid = hid[:, sorted_idx.long()]\n",
    "\n",
    "        y_sorted = self.dropout(self.embed(y_sorted)) # batch_size, output_length, embed_size\n",
    "\n",
    "        packed_seq = nn.utils.rnn.pack_padded_sequence(y_sorted, sorted_len.long().cpu().data.numpy(), batch_first=True)\n",
    "        out, hid = self.rnn(packed_seq, hid)\n",
    "        unpacked, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True)\n",
    "        _, original_idx = sorted_idx.sort(0, descending=False)\n",
    "        output_seq = unpacked[original_idx.long()].contiguous()\n",
    "#         print(output_seq.shape)\n",
    "        hid = hid[:, original_idx.long()].contiguous()\n",
    "\n",
    "        output = F.log_softmax(self.out(output_seq), -1)\n",
    "        \n",
    "        return output, hid\n",
    "    \n",
    "class PlainSeq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder):\n",
    "        super(PlainSeq2Seq, self).__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        \n",
    "    def forward(self, x, x_lengths, y, y_lengths):\n",
    "        encoder_out, hid = self.encoder(x, x_lengths)\n",
    "        output, hid = self.decoder(y=y,\n",
    "                    y_lengths=y_lengths,\n",
    "                    hid=hid)\n",
    "        return output, None\n",
    "    \n",
    "    def translate(self, x, x_lengths, y, max_length=10):\n",
    "        encoder_out, hid = self.encoder(x, x_lengths)\n",
    "        preds = []\n",
    "        batch_size = x.shape[0]\n",
    "        attns = []\n",
    "        for i in range(max_length):\n",
    "            output, hid = self.decoder(y=y,\n",
    "                    y_lengths=torch.ones(batch_size).long().to(y.device),\n",
    "                    hid=hid)\n",
    "            y = output.max(2)[1].view(batch_size, 1)\n",
    "            preds.append(y)\n",
    "            \n",
    "        return torch.cat(preds, 1), None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# masked cross entropy loss\n",
    "class LanguageModelCriterion(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LanguageModelCriterion, self).__init__()\n",
    "\n",
    "    def forward(self, input, target, mask):\n",
    "        # input: (batch_size * seq_len) * vocab_size\n",
    "        input = input.contiguous().view(-1, input.size(2))\n",
    "        # target: batch_size * 1\n",
    "        target = target.contiguous().view(-1, 1)\n",
    "        mask = mask.contiguous().view(-1, 1)\n",
    "        output = -input.gather(1, target) * mask\n",
    "        output = torch.sum(output) / torch.sum(mask)\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "dropout = 0.2\n",
    "hidden_size = 100\n",
    "encoder = PlainEncoder(vocab_size=en_total_words,\n",
    "                      hidden_size=hidden_size,\n",
    "                      dropout=dropout)\n",
    "decoder = PlainDecoder(vocab_size=cn_total_words,\n",
    "                      hidden_size=hidden_size,\n",
    "                      dropout=dropout)\n",
    "model = PlainSeq2Seq(encoder, decoder)\n",
    "model = model.to(device)\n",
    "loss_fn = LanguageModelCriterion().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate(model, data):\n",
    "    model.eval()\n",
    "    total_num_words = total_loss = 0.\n",
    "    with torch.no_grad():\n",
    "        for it, (mb_x, mb_x_len, mb_y, mb_y_len) in enumerate(data):\n",
    "            mb_x = torch.from_numpy(mb_x).to(device).long()\n",
    "            mb_x_len = torch.from_numpy(mb_x_len).to(device).long()\n",
    "            mb_input = torch.from_numpy(mb_y[:, :-1]).to(device).long()\n",
    "            mb_output = torch.from_numpy(mb_y[:, 1:]).to(device).long()\n",
    "            mb_y_len = torch.from_numpy(mb_y_len-1).to(device).long()\n",
    "            mb_y_len[mb_y_len<=0] = 1\n",
    "\n",
    "            mb_pred, attn = model(mb_x, mb_x_len, mb_input, mb_y_len)\n",
    "\n",
    "            mb_out_mask = torch.arange(mb_y_len.max().item(), device=device)[None, :] < mb_y_len[:, None]\n",
    "            mb_out_mask = mb_out_mask.float()\n",
    "\n",
    "            loss = loss_fn(mb_pred, mb_output, mb_out_mask)\n",
    "\n",
    "            num_words = torch.sum(mb_y_len).item()\n",
    "            total_loss += loss.item() * num_words\n",
    "            total_num_words += num_words\n",
    "    print(\"Evaluation loss\", total_loss/total_num_words)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 iteration 0 loss 8.050323486328125\n",
      "Epoch 0 iteration 100 loss 5.278979301452637\n",
      "Epoch 0 iteration 200 loss 4.444733619689941\n",
      "Epoch 0 Training loss 5.433318799975385\n",
      "Evaluation loss 4.822829000278033\n",
      "Epoch 1 iteration 0 loss 4.692166805267334\n",
      "Epoch 1 iteration 100 loss 4.708909511566162\n",
      "Epoch 1 iteration 200 loss 3.8643922805786133\n",
      "Epoch 1 Training loss 4.5993410716009135\n",
      "Epoch 2 iteration 0 loss 4.17959451675415\n",
      "Epoch 2 iteration 100 loss 4.352121829986572\n",
      "Epoch 2 iteration 200 loss 3.5356297492980957\n",
      "Epoch 2 Training loss 4.198561833806036\n",
      "Epoch 3 iteration 0 loss 3.8728413581848145\n",
      "Epoch 3 iteration 100 loss 4.134408950805664\n",
      "Epoch 3 iteration 200 loss 3.303772211074829\n",
      "Epoch 3 Training loss 3.9386860033522813\n",
      "Epoch 4 iteration 0 loss 3.64646053314209\n",
      "Epoch 4 iteration 100 loss 3.947233200073242\n",
      "Epoch 4 iteration 200 loss 3.1333234310150146\n",
      "Epoch 4 Training loss 3.745685762442693\n",
      "Epoch 5 iteration 0 loss 3.481276035308838\n",
      "Epoch 5 iteration 100 loss 3.827454090118408\n",
      "Epoch 5 iteration 200 loss 2.9994454383850098\n",
      "Epoch 5 Training loss 3.5913285724858954\n",
      "Evaluation loss 3.6815984345855037\n",
      "Epoch 6 iteration 0 loss 3.3354697227478027\n",
      "Epoch 6 iteration 100 loss 3.6918392181396484\n",
      "Epoch 6 iteration 200 loss 2.8618223667144775\n",
      "Epoch 6 Training loss 3.465248799091302\n",
      "Epoch 7 iteration 0 loss 3.2224643230438232\n",
      "Epoch 7 iteration 100 loss 3.5980327129364014\n",
      "Epoch 7 iteration 200 loss 2.783277988433838\n",
      "Epoch 7 Training loss 3.357013859409834\n",
      "Epoch 8 iteration 0 loss 3.141510248184204\n",
      "Epoch 8 iteration 100 loss 3.5131657123565674\n",
      "Epoch 8 iteration 200 loss 2.715005397796631\n",
      "Epoch 8 Training loss 3.2614931554428166\n",
      "Epoch 9 iteration 0 loss 3.0618908405303955\n",
      "Epoch 9 iteration 100 loss 3.4437694549560547\n",
      "Epoch 9 iteration 200 loss 2.5995192527770996\n",
      "Epoch 9 Training loss 3.1806184197973404\n",
      "Epoch 10 iteration 0 loss 2.9288880825042725\n",
      "Epoch 10 iteration 100 loss 3.350996971130371\n",
      "Epoch 10 iteration 200 loss 2.5103659629821777\n",
      "Epoch 10 Training loss 3.101915731518774\n",
      "Evaluation loss 3.393061912401112\n",
      "Epoch 11 iteration 0 loss 2.874830722808838\n",
      "Epoch 11 iteration 100 loss 3.3034920692443848\n",
      "Epoch 11 iteration 200 loss 2.4885127544403076\n",
      "Epoch 11 Training loss 3.0369929761565384\n",
      "Epoch 12 iteration 0 loss 2.8056483268737793\n",
      "Epoch 12 iteration 100 loss 3.2505335807800293\n",
      "Epoch 12 iteration 200 loss 2.4071717262268066\n",
      "Epoch 12 Training loss 2.973809002606383\n",
      "Epoch 13 iteration 0 loss 2.7353591918945312\n",
      "Epoch 13 iteration 100 loss 3.178480863571167\n",
      "Epoch 13 iteration 200 loss 2.3422422409057617\n",
      "Epoch 13 Training loss 2.9169208222083847\n",
      "Epoch 14 iteration 0 loss 2.6794426441192627\n",
      "Epoch 14 iteration 100 loss 3.129685401916504\n",
      "Epoch 14 iteration 200 loss 2.3255887031555176\n",
      "Epoch 14 Training loss 2.86419656519231\n",
      "Epoch 15 iteration 0 loss 2.6482393741607666\n",
      "Epoch 15 iteration 100 loss 3.0710315704345703\n",
      "Epoch 15 iteration 200 loss 2.2372782230377197\n",
      "Epoch 15 Training loss 2.8170104509222287\n",
      "Evaluation loss 3.2708830728055336\n",
      "Epoch 16 iteration 0 loss 2.567857503890991\n",
      "Epoch 16 iteration 100 loss 3.0710268020629883\n",
      "Epoch 16 iteration 200 loss 2.238800525665283\n",
      "Epoch 16 Training loss 2.771683479683666\n",
      "Epoch 17 iteration 0 loss 2.5122745037078857\n",
      "Epoch 17 iteration 100 loss 3.002455472946167\n",
      "Epoch 17 iteration 200 loss 2.1964993476867676\n",
      "Epoch 17 Training loss 2.733348611161267\n",
      "Epoch 18 iteration 0 loss 2.49585223197937\n",
      "Epoch 18 iteration 100 loss 2.971094846725464\n",
      "Epoch 18 iteration 200 loss 2.1383423805236816\n",
      "Epoch 18 Training loss 2.6926882812821322\n",
      "Epoch 19 iteration 0 loss 2.436241388320923\n",
      "Epoch 19 iteration 100 loss 2.942230224609375\n",
      "Epoch 19 iteration 200 loss 2.0685524940490723\n",
      "Epoch 19 Training loss 2.6545419067862515\n"
     ]
    }
   ],
   "source": [
    "def train(model, data, num_epochs=20):\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()\n",
    "        total_num_words = total_loss = 0.\n",
    "        for it, (mb_x, mb_x_len, mb_y, mb_y_len) in enumerate(data):\n",
    "            mb_x = torch.from_numpy(mb_x).to(device).long()\n",
    "            mb_x_len = torch.from_numpy(mb_x_len).to(device).long()\n",
    "            mb_input = torch.from_numpy(mb_y[:, :-1]).to(device).long()\n",
    "            mb_output = torch.from_numpy(mb_y[:, 1:]).to(device).long()\n",
    "            mb_y_len = torch.from_numpy(mb_y_len-1).to(device).long()\n",
    "            mb_y_len[mb_y_len<=0] = 1\n",
    "            \n",
    "            mb_pred, attn = model(mb_x, mb_x_len, mb_input, mb_y_len)\n",
    "            \n",
    "            mb_out_mask = torch.arange(mb_y_len.max().item(), device=device)[None, :] < mb_y_len[:, None]\n",
    "            mb_out_mask = mb_out_mask.float()\n",
    "            \n",
    "            loss = loss_fn(mb_pred, mb_output, mb_out_mask)\n",
    "            \n",
    "            num_words = torch.sum(mb_y_len).item()\n",
    "            total_loss += loss.item() * num_words\n",
    "            total_num_words += num_words\n",
    "            \n",
    "            # 更新模型\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            torch.nn.utils.clip_grad_norm_(model.parameters(), 5.)\n",
    "            optimizer.step()\n",
    "            \n",
    "            if it % 100 == 0:\n",
    "                print(\"Epoch\", epoch, \"iteration\", it, \"loss\", loss.item())\n",
    "\n",
    "                \n",
    "        print(\"Epoch\", epoch, \"Training loss\", total_loss/total_num_words)\n",
    "        if epoch % 5 == 0:\n",
    "            evaluate(model, dev_data)\n",
    "train(model, train_data, num_epochs=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BOS you have nice skin . EOS\n",
      "BOS 你 的 皮 膚 真 好 。 EOS\n",
      "你必須吃。\n",
      "\n",
      "BOS you 're UNK correct . EOS\n",
      "BOS 你 部 分 正 确 。 EOS\n",
      "你是一个好的。\n",
      "\n",
      "BOS everyone admired his courage . EOS\n",
      "BOS 每 個 人 都 佩 服 他 的 勇 氣 。 EOS\n",
      "每个人都在学习。\n",
      "\n",
      "BOS what time is it ? EOS\n",
      "BOS 几 点 了 ？ EOS\n",
      "它什么是谁？\n",
      "\n",
      "BOS i 'm free tonight . EOS\n",
      "BOS 我 今 晚 有 空 。 EOS\n",
      "我很快就會。\n",
      "\n",
      "BOS here is your book . EOS\n",
      "BOS 這 是 你 的 書 。 EOS\n",
      "這是你的。\n",
      "\n",
      "BOS they are at lunch . EOS\n",
      "BOS 他 们 在 吃 午 饭 。 EOS\n",
      "他们有个大学。\n",
      "\n",
      "BOS this chair is UNK . EOS\n",
      "BOS 這 把 椅 子 很 UNK 。 EOS\n",
      "這個房間是一個人的。\n",
      "\n",
      "BOS it 's pretty heavy . EOS\n",
      "BOS 它 真 重 。 EOS\n",
      "它是一個好的。\n",
      "\n",
      "BOS many attended his funeral . EOS\n",
      "BOS 很 多 人 都 参 加 了 他 的 葬 礼 。 EOS\n",
      "許多的人都喜歡茶。\n",
      "\n",
      "BOS training will be provided . EOS\n",
      "BOS 会 有 训 练 。 EOS\n",
      "要下雨。\n",
      "\n",
      "BOS someone is watching you . EOS\n",
      "BOS 有 人 在 看 著 你 。 EOS\n",
      "有人是你的。\n",
      "\n",
      "BOS i slapped his face . EOS\n",
      "BOS 我 摑 了 他 的 臉 。 EOS\n",
      "我認為他的手臂。\n",
      "\n",
      "BOS i like UNK music . EOS\n",
      "BOS 我 喜 歡 流 行 音 樂 。 EOS\n",
      "我喜歡打棒球。\n",
      "\n",
      "BOS tom had no children . EOS\n",
      "BOS T o m 沒 有 孩 子 。 EOS\n",
      "汤姆没有人。\n",
      "\n",
      "BOS please lock the door . EOS\n",
      "BOS 請 把 門 鎖 上 。 EOS\n",
      "請把你的車。\n",
      "\n",
      "BOS tom has calmed down . EOS\n",
      "BOS 汤 姆 冷 静 下 来 了 。 EOS\n",
      "汤姆在花園裡。\n",
      "\n",
      "BOS please speak more loudly . EOS\n",
      "BOS 請 說 大 聲 一 點 兒 。 EOS\n",
      "請稍好喝咖啡。\n",
      "\n",
      "BOS keep next sunday free . EOS\n",
      "BOS 把 下 周 日 空 出 来 。 EOS\n",
      "繼續工作很多。\n",
      "\n",
      "BOS i made a mistake . EOS\n",
      "BOS 我 犯 了 一 個 錯 。 EOS\n",
      "我是一個小孩。\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def translate_dev(i):\n",
    "    en_sent = \" \".join([inv_en_dict[w] for w in dev_en[i]])\n",
    "    print(en_sent)\n",
    "    cn_sent = \" \".join([inv_cn_dict[w] for w in dev_cn[i]])\n",
    "    print(\"\".join(cn_sent))\n",
    "\n",
    "    mb_x = torch.from_numpy(np.array(dev_en[i]).reshape(1, -1)).long().to(device)\n",
    "    mb_x_len = torch.from_numpy(np.array([len(dev_en[i])])).long().to(device)\n",
    "    bos = torch.Tensor([[cn_dict[\"BOS\"]]]).long().to(device)\n",
    "\n",
    "    translation, attn = model.translate(mb_x, mb_x_len, bos)\n",
    "    translation = [inv_cn_dict[i] for i in translation.data.cpu().numpy().reshape(-1)]\n",
    "    trans = []\n",
    "    for word in translation:\n",
    "        if word != \"EOS\":\n",
    "            trans.append(word)\n",
    "        else:\n",
    "            break\n",
    "    print(\"\".join(trans))\n",
    "\n",
    "for i in range(100,120):\n",
    "    translate_dev(i)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据全部处理完成，现在我们开始构建seq2seq模型\n",
    "\n",
    "#### Encoder\n",
    "- Encoder模型的任务是把输入文字传入embedding层和GRU层，转换成一些hidden states作为后续的context vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, enc_hidden_size, dec_hidden_size, dropout=0.2):\n",
    "        super(Encoder, self).__init__()\n",
    "        self.embed = nn.Embedding(vocab_size, embed_size)\n",
    "        self.rnn = nn.GRU(embed_size, enc_hidden_size, batch_first=True, bidirectional=True)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.fc = nn.Linear(enc_hidden_size * 2, dec_hidden_size)\n",
    "\n",
    "    def forward(self, x, lengths):\n",
    "        sorted_len, sorted_idx = lengths.sort(0, descending=True)\n",
    "        x_sorted = x[sorted_idx.long()]\n",
    "        embedded = self.dropout(self.embed(x_sorted))\n",
    "        \n",
    "        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, sorted_len.long().cpu().data.numpy(), batch_first=True)\n",
    "        packed_out, hid = self.rnn(packed_embedded)\n",
    "        out, _ = nn.utils.rnn.pad_packed_sequence(packed_out, batch_first=True)\n",
    "        _, original_idx = sorted_idx.sort(0, descending=False)\n",
    "        out = out[original_idx.long()].contiguous()\n",
    "        hid = hid[:, original_idx.long()].contiguous()\n",
    "        \n",
    "        hid = torch.cat([hid[-2], hid[-1]], dim=1)\n",
    "        hid = torch.tanh(self.fc(hid)).unsqueeze(0)\n",
    "\n",
    "        return out, hid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Luong Attention\n",
    "- 根据context vectors和当前的输出hidden states，计算输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Attention(nn.Module):\n",
    "    def __init__(self, enc_hidden_size, dec_hidden_size):\n",
    "        super(Attention, self).__init__()\n",
    "\n",
    "        self.enc_hidden_size = enc_hidden_size\n",
    "        self.dec_hidden_size = dec_hidden_size\n",
    "\n",
    "        self.linear_in = nn.Linear(enc_hidden_size*2, dec_hidden_size, bias=False)\n",
    "        self.linear_out = nn.Linear(enc_hidden_size*2 + dec_hidden_size, dec_hidden_size)\n",
    "        \n",
    "    def forward(self, output, context, mask):\n",
    "        # output: batch_size, output_len, dec_hidden_size\n",
    "        # context: batch_size, context_len, 2*enc_hidden_size\n",
    "    \n",
    "        batch_size = output.size(0)\n",
    "        output_len = output.size(1)\n",
    "        input_len = context.size(1)\n",
    "        \n",
    "        context_in = self.linear_in(context.view(batch_size*input_len, -1)).view(                \n",
    "            batch_size, input_len, -1) # batch_size, context_len, dec_hidden_size\n",
    "        \n",
    "        # context_in.transpose(1,2): batch_size, dec_hidden_size, context_len \n",
    "        # output: batch_size, output_len, dec_hidden_size\n",
    "        attn = torch.bmm(output, context_in.transpose(1,2)) \n",
    "        # batch_size, output_len, context_len\n",
    "\n",
    "        attn.data.masked_fill(mask, -1e6)\n",
    "\n",
    "        attn = F.softmax(attn, dim=2) \n",
    "        # batch_size, output_len, context_len\n",
    "\n",
    "        context = torch.bmm(attn, context) \n",
    "        # batch_size, output_len, enc_hidden_size\n",
    "        \n",
    "        output = torch.cat((context, output), dim=2) # batch_size, output_len, hidden_size*2\n",
    "\n",
    "        output = output.view(batch_size*output_len, -1)\n",
    "        output = torch.tanh(self.linear_out(output))\n",
    "        output = output.view(batch_size, output_len, -1)\n",
    "        return output, attn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decoder\n",
    "- decoder会根据已经翻译的句子内容，和context vectors，来决定下一个输出的单词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, enc_hidden_size, dec_hidden_size, dropout=0.2):\n",
    "        super(Decoder, self).__init__()\n",
    "        self.embed = nn.Embedding(vocab_size, embed_size)\n",
    "        self.attention = Attention(enc_hidden_size, dec_hidden_size)\n",
    "        self.rnn = nn.GRU(embed_size, hidden_size, batch_first=True)\n",
    "        self.out = nn.Linear(dec_hidden_size, vocab_size)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def create_mask(self, x_len, y_len):\n",
    "        # a mask of shape x_len * y_len\n",
    "        device = x_len.device\n",
    "        max_x_len = x_len.max()\n",
    "        max_y_len = y_len.max()\n",
    "        x_mask = torch.arange(max_x_len, device=x_len.device)[None, :] < x_len[:, None]\n",
    "        y_mask = torch.arange(max_y_len, device=x_len.device)[None, :] < y_len[:, None]\n",
    "        mask = (1 - x_mask[:, :, None] * y_mask[:, None, :]).byte()\n",
    "        return mask\n",
    "    \n",
    "    def forward(self, ctx, ctx_lengths, y, y_lengths, hid):\n",
    "        sorted_len, sorted_idx = y_lengths.sort(0, descending=True)\n",
    "        y_sorted = y[sorted_idx.long()]\n",
    "        hid = hid[:, sorted_idx.long()]\n",
    "        \n",
    "        y_sorted = self.dropout(self.embed(y_sorted)) # batch_size, output_length, embed_size\n",
    "\n",
    "        packed_seq = nn.utils.rnn.pack_padded_sequence(y_sorted, sorted_len.long().cpu().data.numpy(), batch_first=True)\n",
    "        out, hid = self.rnn(packed_seq, hid)\n",
    "        unpacked, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True)\n",
    "        _, original_idx = sorted_idx.sort(0, descending=False)\n",
    "        output_seq = unpacked[original_idx.long()].contiguous()\n",
    "        hid = hid[:, original_idx.long()].contiguous()\n",
    "\n",
    "        mask = self.create_mask(y_lengths, ctx_lengths)\n",
    "\n",
    "        output, attn = self.attention(output_seq, ctx, mask)\n",
    "        output = F.log_softmax(self.out(output), -1)\n",
    "        \n",
    "        return output, hid, attn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Seq2Seq\n",
    "- 最后我们构建Seq2Seq模型把encoder, attention, decoder串到一起"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder):\n",
    "        super(Seq2Seq, self).__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        \n",
    "    def forward(self, x, x_lengths, y, y_lengths):\n",
    "        encoder_out, hid = self.encoder(x, x_lengths)\n",
    "        output, hid, attn = self.decoder(ctx=encoder_out, \n",
    "                    ctx_lengths=x_lengths,\n",
    "                    y=y,\n",
    "                    y_lengths=y_lengths,\n",
    "                    hid=hid)\n",
    "        return output, attn\n",
    "    \n",
    "    def translate(self, x, x_lengths, y, max_length=100):\n",
    "        encoder_out, hid = self.encoder(x, x_lengths)\n",
    "        preds = []\n",
    "        batch_size = x.shape[0]\n",
    "        attns = []\n",
    "        for i in range(max_length):\n",
    "            output, hid, attn = self.decoder(ctx=encoder_out, \n",
    "                    ctx_lengths=x_lengths,\n",
    "                    y=y,\n",
    "                    y_lengths=torch.ones(batch_size).long().to(y.device),\n",
    "                    hid=hid)\n",
    "            y = output.max(2)[1].view(batch_size, 1)\n",
    "            preds.append(y)\n",
    "            attns.append(attn)\n",
    "        return torch.cat(preds, 1), torch.cat(attns, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dropout = 0.2\n",
    "embed_size = hidden_size = 100\n",
    "encoder = Encoder(vocab_size=en_total_words,\n",
    "                       embed_size=embed_size,\n",
    "                      enc_hidden_size=hidden_size,\n",
    "                       dec_hidden_size=hidden_size,\n",
    "                      dropout=dropout)\n",
    "decoder = Decoder(vocab_size=cn_total_words,\n",
    "                      embed_size=embed_size,\n",
    "                      enc_hidden_size=hidden_size,\n",
    "                       dec_hidden_size=hidden_size,\n",
    "                      dropout=dropout)\n",
    "model = Seq2Seq(encoder, decoder)\n",
    "model = model.to(device)\n",
    "loss_fn = LanguageModelCriterion().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 iteration 0 loss 8.078022003173828\n",
      "Epoch 0 iteration 100 loss 5.414377689361572\n",
      "Epoch 0 iteration 200 loss 4.643333435058594\n",
      "Epoch 0 Training loss 5.485134587536152\n",
      "Evaluation loss 5.067514630874862\n",
      "Epoch 1 iteration 0 loss 4.940210342407227\n",
      "Epoch 1 iteration 100 loss 4.9903435707092285\n",
      "Epoch 1 iteration 200 loss 4.186498641967773\n",
      "Epoch 1 Training loss 4.877356682952294\n",
      "Epoch 2 iteration 0 loss 4.509239196777344\n",
      "Epoch 2 iteration 100 loss 4.570853233337402\n",
      "Epoch 2 iteration 200 loss 3.7934508323669434\n",
      "Epoch 2 Training loss 4.453642889638262\n",
      "Epoch 3 iteration 0 loss 4.11014986038208\n",
      "Epoch 3 iteration 100 loss 4.230580806732178\n",
      "Epoch 3 iteration 200 loss 3.4451844692230225\n",
      "Epoch 3 Training loss 4.105205834096106\n",
      "Epoch 4 iteration 0 loss 3.788179397583008\n",
      "Epoch 4 iteration 100 loss 3.984476089477539\n",
      "Epoch 4 iteration 200 loss 3.205059289932251\n",
      "Epoch 4 Training loss 3.8313639103406314\n",
      "Epoch 5 iteration 0 loss 3.572876214981079\n",
      "Epoch 5 iteration 100 loss 3.7907521724700928\n",
      "Epoch 5 iteration 200 loss 3.0604655742645264\n",
      "Epoch 5 Training loss 3.61275750220716\n",
      "Evaluation loss 3.6225900108158475\n",
      "Epoch 6 iteration 0 loss 3.331376552581787\n",
      "Epoch 6 iteration 100 loss 3.607234239578247\n",
      "Epoch 6 iteration 200 loss 2.8438034057617188\n",
      "Epoch 6 Training loss 3.4240881394610914\n",
      "Epoch 7 iteration 0 loss 3.1553823947906494\n",
      "Epoch 7 iteration 100 loss 3.4283368587493896\n",
      "Epoch 7 iteration 200 loss 2.679870367050171\n",
      "Epoch 7 Training loss 3.2619650765874195\n",
      "Epoch 8 iteration 0 loss 3.0175576210021973\n",
      "Epoch 8 iteration 100 loss 3.313087224960327\n",
      "Epoch 8 iteration 200 loss 2.573970079421997\n",
      "Epoch 8 Training loss 3.119750910546451\n",
      "Epoch 9 iteration 0 loss 2.8687644004821777\n",
      "Epoch 9 iteration 100 loss 3.2016961574554443\n",
      "Epoch 9 iteration 200 loss 2.4501001834869385\n",
      "Epoch 9 Training loss 2.9937007481445184\n",
      "Epoch 10 iteration 0 loss 2.7964212894439697\n",
      "Epoch 10 iteration 100 loss 3.094231128692627\n",
      "Epoch 10 iteration 200 loss 2.2865397930145264\n",
      "Epoch 10 Training loss 2.879919764606877\n",
      "Evaluation loss 3.164760209368642\n",
      "Epoch 11 iteration 0 loss 2.6683473587036133\n",
      "Epoch 11 iteration 100 loss 3.008727788925171\n",
      "Epoch 11 iteration 200 loss 2.1880834102630615\n",
      "Epoch 11 Training loss 2.7794466071573467\n",
      "Epoch 12 iteration 0 loss 2.5640454292297363\n",
      "Epoch 12 iteration 100 loss 2.896376132965088\n",
      "Epoch 12 iteration 200 loss 2.1036128997802734\n",
      "Epoch 12 Training loss 2.684113484535982\n",
      "Epoch 13 iteration 0 loss 2.520007371902466\n",
      "Epoch 13 iteration 100 loss 2.8189423084259033\n",
      "Epoch 13 iteration 200 loss 2.0698890686035156\n",
      "Epoch 13 Training loss 2.5990255668547055\n",
      "Epoch 14 iteration 0 loss 2.42832612991333\n",
      "Epoch 14 iteration 100 loss 2.7819204330444336\n",
      "Epoch 14 iteration 200 loss 1.923954725265503\n",
      "Epoch 14 Training loss 2.5176252404633574\n",
      "Epoch 15 iteration 0 loss 2.360988140106201\n",
      "Epoch 15 iteration 100 loss 2.6843974590301514\n",
      "Epoch 15 iteration 200 loss 1.912152886390686\n",
      "Epoch 15 Training loss 2.4463321701504275\n",
      "Evaluation loss 2.9698491313827047\n",
      "Epoch 16 iteration 0 loss 2.2877912521362305\n",
      "Epoch 16 iteration 100 loss 2.6055469512939453\n",
      "Epoch 16 iteration 200 loss 1.8231658935546875\n",
      "Epoch 16 Training loss 2.3756549535366713\n",
      "Epoch 17 iteration 0 loss 2.191697597503662\n",
      "Epoch 17 iteration 100 loss 2.5865063667297363\n",
      "Epoch 17 iteration 200 loss 1.7817124128341675\n",
      "Epoch 17 Training loss 2.313343924902058\n",
      "Epoch 18 iteration 0 loss 2.1245803833007812\n",
      "Epoch 18 iteration 100 loss 2.525496482849121\n",
      "Epoch 18 iteration 200 loss 1.672200322151184\n",
      "Epoch 18 Training loss 2.2498218108556114\n",
      "Epoch 19 iteration 0 loss 2.06477427482605\n",
      "Epoch 19 iteration 100 loss 2.443316698074341\n",
      "Epoch 19 iteration 200 loss 1.6326298713684082\n",
      "Epoch 19 Training loss 2.19988960411091\n",
      "Epoch 20 iteration 0 loss 2.0234487056732178\n",
      "Epoch 20 iteration 100 loss 2.416968822479248\n",
      "Epoch 20 iteration 200 loss 1.583616852760315\n",
      "Epoch 20 Training loss 2.1513965044733827\n",
      "Evaluation loss 2.8699020465835643\n",
      "Epoch 21 iteration 0 loss 2.008730411529541\n",
      "Epoch 21 iteration 100 loss 2.3642444610595703\n",
      "Epoch 21 iteration 200 loss 1.5385680198669434\n",
      "Epoch 21 Training loss 2.098746986360735\n",
      "Epoch 22 iteration 0 loss 1.910429835319519\n",
      "Epoch 22 iteration 100 loss 2.339489459991455\n",
      "Epoch 22 iteration 200 loss 1.4784246683120728\n",
      "Epoch 22 Training loss 2.051404798098097\n",
      "Epoch 23 iteration 0 loss 1.8959044218063354\n",
      "Epoch 23 iteration 100 loss 2.2653536796569824\n",
      "Epoch 23 iteration 200 loss 1.4792706966400146\n",
      "Epoch 23 Training loss 2.00636701965731\n",
      "Epoch 24 iteration 0 loss 1.8477107286453247\n",
      "Epoch 24 iteration 100 loss 2.1904118061065674\n",
      "Epoch 24 iteration 200 loss 1.3925689458847046\n",
      "Epoch 24 Training loss 1.965628425139225\n",
      "Epoch 25 iteration 0 loss 1.7790645360946655\n",
      "Epoch 25 iteration 100 loss 2.182420492172241\n",
      "Epoch 25 iteration 200 loss 1.3576843738555908\n",
      "Epoch 25 Training loss 1.9238889035465652\n",
      "Evaluation loss 2.826008448512912\n",
      "Epoch 26 iteration 0 loss 1.73543381690979\n",
      "Epoch 26 iteration 100 loss 2.1740329265594482\n",
      "Epoch 26 iteration 200 loss 1.328704595565796\n",
      "Epoch 26 Training loss 1.889945533318946\n",
      "Epoch 27 iteration 0 loss 1.7498269081115723\n",
      "Epoch 27 iteration 100 loss 2.1384894847869873\n",
      "Epoch 27 iteration 200 loss 1.277467966079712\n",
      "Epoch 27 Training loss 1.852515173441663\n",
      "Epoch 28 iteration 0 loss 1.6980342864990234\n",
      "Epoch 28 iteration 100 loss 2.1195883750915527\n",
      "Epoch 28 iteration 200 loss 1.2595137357711792\n",
      "Epoch 28 Training loss 1.8210893462516964\n",
      "Epoch 29 iteration 0 loss 1.6773594617843628\n",
      "Epoch 29 iteration 100 loss 2.0760860443115234\n",
      "Epoch 29 iteration 200 loss 1.2345834970474243\n",
      "Epoch 29 Training loss 1.7873437400435428\n"
     ]
    }
   ],
   "source": [
    "train(model, train_data, num_epochs=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BOS you have nice skin . EOS\n",
      "BOS 你 的 皮 膚 真 好 。 EOS\n",
      "你好害怕。\n",
      "\n",
      "BOS you 're UNK correct . EOS\n",
      "BOS 你 部 分 正 确 。 EOS\n",
      "你是全子的声音。\n",
      "\n",
      "BOS everyone admired his courage . EOS\n",
      "BOS 每 個 人 都 佩 服 他 的 勇 氣 。 EOS\n",
      "他的袋子是他的勇氣。\n",
      "\n",
      "BOS what time is it ? EOS\n",
      "BOS 几 点 了 ？ EOS\n",
      "多少时间是什么？\n",
      "\n",
      "BOS i 'm free tonight . EOS\n",
      "BOS 我 今 晚 有 空 。 EOS\n",
      "我今晚有空。\n",
      "\n",
      "BOS here is your book . EOS\n",
      "BOS 這 是 你 的 書 。 EOS\n",
      "这儿是你的书。\n",
      "\n",
      "BOS they are at lunch . EOS\n",
      "BOS 他 们 在 吃 午 饭 。 EOS\n",
      "他们在午餐。\n",
      "\n",
      "BOS this chair is UNK . EOS\n",
      "BOS 這 把 椅 子 很 UNK 。 EOS\n",
      "這些花一下是正在的。\n",
      "\n",
      "BOS it 's pretty heavy . EOS\n",
      "BOS 它 真 重 。 EOS\n",
      "它很美的脚。\n",
      "\n",
      "BOS many attended his funeral . EOS\n",
      "BOS 很 多 人 都 参 加 了 他 的 葬 礼 。 EOS\n",
      "多多衛年轻地了他。\n",
      "\n",
      "BOS training will be provided . EOS\n",
      "BOS 会 有 训 练 。 EOS\n",
      "别将被付錢。\n",
      "\n",
      "BOS someone is watching you . EOS\n",
      "BOS 有 人 在 看 著 你 。 EOS\n",
      "有人看你。\n",
      "\n",
      "BOS i slapped his face . EOS\n",
      "BOS 我 摑 了 他 的 臉 。 EOS\n",
      "我把他的臉抱歉。\n",
      "\n",
      "BOS i like UNK music . EOS\n",
      "BOS 我 喜 歡 流 行 音 樂 。 EOS\n",
      "我喜歡音樂。\n",
      "\n",
      "BOS tom had no children . EOS\n",
      "BOS T o m 沒 有 孩 子 。 EOS\n",
      "汤姆没有照顧孩子。\n",
      "\n",
      "BOS please lock the door . EOS\n",
      "BOS 請 把 門 鎖 上 。 EOS\n",
      "请把門開門。\n",
      "\n",
      "BOS tom has calmed down . EOS\n",
      "BOS 汤 姆 冷 静 下 来 了 。 EOS\n",
      "汤姆在做了。\n",
      "\n",
      "BOS please speak more loudly . EOS\n",
      "BOS 請 說 大 聲 一 點 兒 。 EOS\n",
      "請說更多。\n",
      "\n",
      "BOS keep next sunday free . EOS\n",
      "BOS 把 下 周 日 空 出 来 。 EOS\n",
      "繼續下週一下一步。\n",
      "\n",
      "BOS i made a mistake . EOS\n",
      "BOS 我 犯 了 一 個 錯 。 EOS\n",
      "我做了一件事。\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(100,120):\n",
    "    translate_dev(i)\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
